While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by 2-20 times over naive Monte Carlo sampling methods and 10-300P times (where P is the number of processors) over real-world testing.

Very strong paper which is reporting on a large scale AV project. As such it describes many parts of a very complex system, relying on a lot of other work and systems, but contributing by showing how it all fits together to do a new thing. The new thing is the use of importance sampling over parameters for models of human drivers in a driving simulator, in order to better estimate collision probabilities for AV controllers. This relies on a huge stack of prior work comprising: (1) a complex physical driving simulator (which brilliantly, is open sourced, and presentation of such a facility to the community would make a great publication just by itself); and (2) a parametric model of human driver behaviour based on a GAN-style imitation learner, which maps an input vector describing the scene to a prediction of other drivers actions. Paper is clearly written and structured, however for slightly dumber readers like this one I would suggest trying to add a bit more detail on exactly what are the inputs and outputs of this human driver predictor.

While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by 2-20 times over naive Monte Carlo sampling methods and 10-300P times (where P is the number of processors) over real-world testing.